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AUGMENTET REALITY APPARATUS AND CT METHOD 



FIELD OF THE INVENTION 

The present invention relates to surgical instruments, and, more 
particularly, to an image overlay system to assist surgical interventions inside 
conventiona.1 cross-sectional imaging systems. 

BACKGROUND OF THE INVENTION 

Multiple solutions have been developed to track surgical devices and 
combine auxiliary information with medical image data on a computer screen to 
assist surgical interventions. Since the end of the 1980's, significant research 
has been devoted to three-dimensional ("3D") medical images for surgical 
simulation and planning. Ordinarily, these 3D images are shown on a flat two- 
dimensional ("2D") display on the wall of an operating room. The fundamental 
drawback of these systems is that the operating surgeon must look away from 
the operation field, observe the display, then turn his or her attention back to 
the operation field and use hand-eye coordination to navigate a surgical device. 
In this arrangement, the surgeon must mentally register the computer generated 
2D image with the anatomy of the patient seen in the surgical field, and use 
hand-eye coordination to execute the surgical plan. Therefore, the actions of 
the surgeon are not truly objective, primarily because the images and the patient 
are not accurately registered together. 

To achieve better comprehension of 3D medical images, augmented 
reality displays are becoming more popular in the medical field. For example, 
one research group named the Roberts' Group has developed a frameless, 
image-overlay system for a stereoscopic microscope for neurosurgery. Another 
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research group named the Fuchs' Group has developed an augmented reality 
system under ultrasound image guidance therapy. In these systems, medical 
images are visualized using a stereoscopic display, e.g., a polarized shutter 
system with two small monitors located on the user's head. However, there are 
potential errors in alignment and spatial registration that can occur in using this 
binocular system approach. 

Other known systems have different problems that diminish their ability 
to be used with medical image data to assist surgical interventions. 

DiGioia and others have developed a system that requires cumbersome 
calibration and both the patient and operating surgeon to be instrumented with 
tracking devices to ensure proper registration of image, patient, and viewer at 
all times. 

Masamune and others have developed a stand-alone 2D image overlay 
mechanism for CT and MRI images, but with no physical attachment to the 
imaging device. The perceived location of the overlay image is independent of 
the viewer's location. Masumune's system is not intra-operative and it requires 
post-imaging registration between the patient and the image overlay device. 
The system uses multiple image slices and modalities, but it does not contain a 
surgical placement mechanism. 

Stetten and others have developed a 2D image overlay system coupled 
with an ultrasonic transducer. The ultrasonic transducer is rigidly fixed with 
respect to the image overlay device to preserve registration. The system 
provides real-time imagery, but it does not allow for multiple image directions 
and locations, and it does not include a surgical instrument placement 
mechanism. 

Each of the currently known systems is affected by one or more of the 
following problems: (1) the patient and surgeon are both instrumented with 
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tracking devices; (2) there is a lack of ability to match intra-operative images 
into the physical patient; (3) they require cumbersome calibration and quality 
control; (4) they have a lack of an ability to use arbitrary image planes; and 
(5)they lack a surgical device placement mechanism. 

SUMMARY OF THE INVENTION 

The present invention solves the foregoing problems by displaying 
patient slice images so that they appear to be physically positioned with respect 
to the patient at the point at which they were taken and by facilitating the 
placement of surgical devices into the patient via a surgical instrument 
placement mechanism. 

Accordingly, it is an object of the present invention to provide an intra- 
operative visualization system that allows simultaneous viewing of the surgical 
field as seen by the surgeon and the patient's internal anatomy. It is also an 
object of the present invention to provide an image overlay system that 
provides accurate depth perception and overlay of image slices on the patient's 
body. It is a further object of the present invention to provide a system that 
makes a CT image slice "appear" in the exact same location where it was 
acquired, while the patient is still in the CT scanner. 

The present invention is a simple, safe and inexpensive image overlay 
system to assist surgical interventions inside a conventional CT scanner or 
other cross-sectional imaging system, such as a magnetic resonance imaging 
("MRI") system, positron emission tomography ("PET"), or single photon 
emission computed tomography ("SPECT") system. The overlay system of the 
present invention is mounted non-invasively on the gantry of the CT scanner. 
The image overlay system includes a flat liquid crystal display ("LCD"), a half- 
mirror, and a passive holding attachment that supports the display and mirror 
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and provides seven degrees-of -freedom for proper placement of the display and 
mirror in the CT scanner or other cross-sectional imaging system scanner. The 
image overlay system also includes a mechanism to facilitate placement and 
insertion into a patient of a surgical device. 

A pre-operative calibration process is used to spatially register the 
display, half-mirror, and imaging plane of the scanner using a rigid calibration 
object. Following the calibration, the patient is brought into the scanner, and 
images acquired are sent to the overlay display. Looking at the patient through 
the half-mirror, the CT slice image appears to be floating inside the patient in 
correct size and position. This vision enables the surgeon to see both the 
surface and the inside of the patient at the same time, and can be used by the 
surgeon in guiding a surgical intervention. Surgical devices are navigated in 
the visual field of view. Invasive devices are visible up to the point they enter 
the body. If the physician wishes to see the portion of the device that has been 
inserted into the body, then a new image slice needs to be acquired and 
displayed on the overlay, provided the given imaging modality is able to 
visualize the surgical device. (For example, CT scanner can visualize metal 
needles.) The perceived location of the overlay image is independent of the 
viewer's location. Thus, the surgeon and patient are not instrumented with any 
type of tracking devices. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIGURE 1 is a schematic drawing of the basic physical layout of the 
image overlay system of the present invention. 

FIGURE 2 is a schematic drawing of the image overlay system of the 
present invention in which multiple CT slices are obtained by vertical 
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translation of the display to compensate for organs that are larger than one 
slice. 

FIGURE 3 is a schematic drawing of the image overlay system of the 
present invention in which multiple CT slices are obtained by horizontal 
translation of the mirror to compensate for organs that are larger than one slice. 

FIGURE 4 is a schematic drawing of the image overlay system of the 
present invention in which image panning by horizontal translation of the 
display is achieved to compensate for a body that is too large to fit on the 
display. 

FIGURE 5 is a schematic drawing of the image overlay system of the 
present invention in which patient interference with the system is resolved by a 
coordinated tilting of the mirror and the display. 

FIGURE 6 is a schematic drawing of the image overlay system of the 
present invention in which tilted CT slice images are obtained using a tilting of 
the CT gantry. 

FIGURE 7 is a schematic drawing of the image overlay system of the 
present invention in which an overlay of synthetic (re-sampled) angled CT 
slices are obtained by tilting the display. 

FIGURE 8 is a schematic drawing of the image overlay system of the 
present invention in which an overlay of synthetic (re-sampled) angled CT 
slices are obtained by tilting the mirror. 

FIGURE 9 is a schematic drawing of the image overlay system of the 
present invention in which a robot surgical device placement mechanism is 
remotely controlled by a joystick control. 

FIGURE 10 is a schematic drawing of the image overlay system of the 
present invention in which a robot surgical device placement mechanism is 
remotely controlled by a personal computer. 
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FIGURE 1 1 is a schematic drawing of the image overlay system of the 
present invention in which the surgical device placement mechanism is a 
mechanical device that is manually actuated by a surgeon. 

FIGURE 12 is a schematic drawing of the image overlay system of the 
present invention in which the surgical device is placed by the surgeons free 
hand. 

FIGURE 13 is a schematic drawing of the image overlay system of the 
present invention in which a tracking device is used to assist the surgeon in 
placement of the surgical device by free hand. 

FIGURE 14 is a schematic drawing of the image overlay system of the 
present invention in which a tracking device is used with a mechanical surgical 
device placement mechanism that is manually actuated by the surgeon. 

FIGURE 15 is a schematic drawing of the image overlay system of the 
present invention in which a tracking device transmits positioning information 
to be visually superimposed on the virtual overlay image to assist the surgeon 
in placing a surgical device by free hand. 

FIGURE 16 is a schematic drawing of the image overlay system of the 
present invention in which a tracking device transmits positioning information 
to be visually superimposed on the virtual overlay image to assist the surgeon 
in placing a surgical device using a mechanical placement device. 

FIGURE 17 is a schematic drawing of the image overlay system of the 
present invention in which data from an encoded instrument placement 
mechanism is visually superimposed on the virtual overlay image to assist the 
surgeon in placing a surgical device using the placement mechanism. 

FIGURE 1 8 is a schematic drawing of the image overlay system of the 
present invention in which data from the computed tomography scanner is 
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registered with multi-modal pre-operative images so that a composite image is 
sent to the image overlay system's display. 

FIGURE 19 is a schematic drawing of the image overlay system of the 
present invention showing a first step in an image overlay calibration 
procedure. 

FIGURE 20 is a schematic drawing of the image overlay system of the 
present invention showing a second step in an image overlay calibration 
procedure. 

FIGURE 21 is a schematic drawing of the image overlay system of the 
present invention showing a third step in an image overlay calibration 
procedure. 

FIGURE 22 is a schematic drawing of the image overlay system of the 
present invention using passive calibration for placement of the surgical device 
mechanism. 

FIGURE 23 is a schematic drawing of the image overlay system of the 
present invention using active calibration for placement of the surgical device 
mechanism. 

FIGURE 24 is a schematic drawing of the basic physical layout of the 
image overlay system of the present invention shown in FIGURE 1, except that 
the image display has an orientation other than the horizontal orientation shown 
in FIGURE 1. 

FIGURE 25 is a perspective view of the image overlay system of the 
present invention shown in FIGURE 24. 

FIGURE 26 is a perspective view of a holding attachment with seven 
degrees-of-freedom used to mount the image display in the arrangement of the 
image overlay system shown in FIGURE 24. 
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FIGURE 27 is a perspective view of a triangle marker used in a system 
registration procedure positioned on the scanner bed of the CT scanner and 
adjusted to the laser of the CT scanner. 

FIGURE 28 is a computer generated image of a slice of the triangle 
marker taken by the CT scanner. 

FIGURE 29 is a perspective view of the triangle marker used in the 
system registration procedure with the overlay image of the slice of the triangle 
marker taken by the CT scanner being viewed through the half transparent 
mirror. 

DESCRIPTION OF THE PREFERRED EMBODIMENT 

FIGURE 1 is a schematic drawing of the basic layout of the image 
overlay system 10 of the present invention. The system 10 is designed to be 
used with a computed tomography ("CT") scanner system 1 1 or other cross- 
sectional imaging system, such as a magnetic resonance imaging ("MRI") 
system, positron emission tomography ("PET"), or single photon emission 
computed tomography ("SPECT") system. The system can also be used with 
ultrasound and thermal imaging systems. By way of example, the present 
invention will be described with respect to a CT system, which typically 
includes a CT gantry 12 and a CT table 14. The system of the present 
invention allows a surgeon 18 to more precisely place a surgical device 32 in a 
patient 16 laying on CT table 14 using a CT slice image of patient 16 for 
guidance. 

Image overlay system 10 includes a display 22 for displaying the slice 
image of patient 16 obtained by CT scanner system 1 1, a half transparent mirror 
24 for reflecting the slice image shown on display 22, a surgical device 
placement mechanism 30 to assist in the placement of surgical device 32 in 
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patient 16, and a mounting device 26 mounted on top of CT gantry 12. 
Mounting device 26 is used to support display 22, mirror 24 and placement 
mechanism 30. Half- mirror 24 is preferably either a half silver mirror or a 
light brown acrylic plate. 

CT scanner system 1 1 has a CT image plane 20 in which slice images of 
patient 16 are taken. In the embodiment of the image overlay system 10 shown 
in FIGURE 1, mirror 24 is oriented with respect to image plane 20 at an angle 
25 that is nominally 45°. Display 22 is then oriented with respect to mirror 24 
at an angle 23 that is the same as angle 25, and thus, nominally 45°. 

Surgical device placement mechanism 30 shown in the embodiment of 
FIGURE 1 is a mechanical device that constrains surgical device 32 in a plane 
that coincides with the laser marker 21 of CT gantry 12. Preferably, placement 
mechanism 30 is designed to allow surgical device 32 to be moved in the x and 
y directions of the constrained plane and rotated within the same plane so that it 
can be properly inserted in patient 16. Preferably, placement mechanism 30 
includes a plurality of arms that are under tension for support and that are 
interconnected by a plurality of pivot joints. One example of such a mechanism 
is a surgical tool support used by dentists, but that is constrained to move in a 
selected plane. Surgical instrument 32 can be a needle or other type of surgical 
instrument, such as a drill, trocar, biopsy needle, injection device or tissue 
ablator, by way of example. 

FIGURE 24 is another drawing of the basic layout of image overlay 
system 10 of the present invention, except that in FIGURE 24 the slice image 
display 22 has a tilted orientation that is different from the generally horizontal 
orientation of display 22 shown in FIGURE 1. The half transparent mirror 24 
also shown in FIGURE 24 thus has a different orientation and position from the 
position and orientation of mirror 24 shown in FIGURE 1 to accommodate the 
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change in display 22's orientation. FIGURE 25 is a perspective view of the 
embodiment of image overlay system 10 shown in FIGURE 24. The relative 
angle between display 22 and mirror 24 is not the nominal 45 degrees. The 
display 22 and mirror 24 were tilted, in order to give sufficient room for the 
patient and sufficient workspace for the surgeon. This solution is further 
explained later in Figure 5. In the embodiment of image overlay system 10 
shown in FIGURES 24 and 25, mounting device 26 is detachably placed over 
CT gantry 12 as in the embodiment shown in FIGURE 1. In addition, attached 
to the mounting device 26 shown in FIGURES 24 and 25 is a holding 
attachment 78 on which are mounted display 22 and half -mirror 24. 

As seen in FIGURE 26, holding attachment 78 includes a u-shaped 
bracket 80 that is positioned over a cross brace 81 of mounting device 26. 
Canterlevered from bracket 80 is a shaft 82 on which a parallel link 84 is 
rotatably mounted at a first pivot joint 86. At the opposite end of parallel link. 
84 is a second pivot joint 88 to which is connected a third pivot joint 92 
supporting a mounting plate 94 on which display 22 is mounted. Included in 
mounting plate 94 is a circular groove 90 in which is rotatably mounted a cross 
arm 96 forming part of a frame 76 that supports display 22 and half-mirror 24. 
Mounted at the ends of cross arm 96 are two vertically oriented arms 97 on 
which mirror 24 is rotatably mounted. As can be seen in FIGURE 26, holding 
attachment 78 provides seven degrees of freedom (indicated by arrows) for the 
placement and orientation of display 22 and half-mirror 24 with respect to a 
patient 16 lying on CT table 14 of CT scanner system 11. 

In current CT imaging systems, an image slice is typically taken at the 
center of the gantry. Holding attachment 78 allows mirror 24 of the overlay 
system 10 to be accurately positioned on the center of CT gantry 12, without 
making alterations to CT scanner 11 in violation of safety regulations. Holding 
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attachment 78 also allows the overlay system 10 to be easily installed on a 
variety of CT gantries of different shapes. 

Registration of the CT image taken by CT scanner system 1 1 and the 
patient 16 is critical to proper operation of the image overlay system 10. 
FIGURES 27 to 29 describe the preferred registration method used with the 
present invention. Referring first to FIGURE 27, a triangle-shaped fiducial 
marker 100 is placed on CT table 14 and adjusted to the laser marker 21 of CT 
scanner system 1 1 which shows the plane of acquisition of system 11. As can 
be seen in FIGURE 27, triangle marker 100 is made from a clear material and 
has a longitudinal dimension that allows a slice image of marker 100 to be 
taken by scanner 11. After a slice image of triangle marker 100 is taken, the 
image is transferred, preferably via DICOM, to a personal computer 50 
(FIGURE 24) used for image processing. The resolution of display 22 is then 
pre-calibrated to obtain a full sized image, after which the slice image is 
transferred to computer 50 and adjusted to its full size. The image is then sent 
to display 22 and adjusted while looking through mirror 24 to the real image of 
triangle marker 100 using the edges of triangle marker 100 and surgeon 18's 
perception. Upon completing the registration procedure, it is possible to 
observe the slice image of marker 100 in an accurate position. The calibration 
object triangle marker 100 is then replaced with patient 16, while making sure 
that mirror 24 and display 22 are not moved. 

Referring now to FIGURE 25, the image overlay system 10 is designed 
to make a CT slice image 75 appear in the exact same location where it was 
acquired from patient 16, while patient 16 is still in CT scanner 1 1. By 
superimposing image 75 on patient 16, the physician 18 can see the surface of 
patient 16 and the inside of the body of patient 16 at the same time. This helps 
the physician 1 8 to find the optimal access to a target within patient 16, while 
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minimizing collateral damage to normal, healthy structures around the target. 
Slice image 75 is not projected into the surgical field. It is merely made to 
appear as though it is there. Thus, the central piece of system 10 is half-mirror 
24. Half-mirror 24 allows the scene in the surgical field behind it to be seen at 
the same time as reflections in half-mirror 24. The reflections are fainter than 
they would be in a full mirror and the transparent surgical field scene is also 
clearer through clear glass, but with proper lighting, a reasonable compromise 
is achieved between the two. Optimal lighting can be determined 
experimentally, by using a simple continuous light dimmer attached to the light 
switch of the room. In operation, CT scanner 1 1 acquires a CT slice image of 
patient 16. Slice image 75 is rendered on display 22 which preferably is a 
distortion-free flat panel display. Display 22 is positioned above half-mirror 
24, so that the reflection of slice image 75 coincides with patient 16's body seen 
behind mirror 24. Given the previous registration of the real view and the slice 
image 75 in mirror 24 using triangle i00, the result is an optically stable 
alignment of the real view and the slice image 75 in mirror 24, provided that 
mirror 24, display 22, and scanner 1 1 have been properly aligned. 

Referring now to FIGURES 2 and 3, depicted therein are two schemes 
for using multiple CT slice images to view organs that are larger than a single 
CT slice image without physically moving patient 16. In FIGURE 2, display 22 
is moved some vertical distance d from its original position, either toward or 
away from patient 16, to a new position. As a consequence, another overlay 
image 28A is displayed a horizontal distance d away (commensurate with the 
change in position of display 22) from the position at which the original overlay 
image 28 was displayed. This allows a second CT slice image of patient 16 
taken at a position different from where the original image slice was taken to be 
displayed at the proper location with respect to patient 16. 
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In FIGURE 3, half-mirror 24 is moved some horizontal distance d from 
its original position, either toward or away from patient 16, to a new position. 
As a consequence, second overlay image 28A is displayed at a new position a 
horizontal distance d away (commensurate with the change in position of half- 
mirror 24) from the position at which original overlay image 28 was displayed. 
Here again, this allows the second CT slice image of patient 16 to be displayed 
at the proper location with respect to patient 16. 

FIGURE 4 discloses a scheme for image panning by parallel translation 
of display 22 or by moving the slice image on display 22 via software control to 
accommodate a body that is too large to fit on display 22. In this arrangement, 
display 22 is translated in a horizontal plane a distance d. The result is that 
overlay image 28 is translated in a vertical plane by the same distance d as 
display 22 is moved. 

FIGURE 5 shows an arrangement for resolving interference of the image 
overlay system 10 with a patient 16. In this arrangement, half-mirror 24 is 
tilted so as to form with respect to image plane 20 an angle 25 A that is equal to 
some angle ct°. When half-mirror 24 is tilted to eliminate interference with 
patient 16, the angular relationship between half-mirror 24 and display 22 must 
be maintained. Thus, the angular relationship between display 22 and half- 
mirror 24 must also be an angle 23A that is equal to oc°. 

FIGURE 6 discloses an arrangement for using tilted CT slice images that 
are obtained from patient 16 by tilting CT gantry 12. Tilting the CT gantry is 
achieved by a standard mechanism commonly available on contemporary CT 
scanners. This feature is not typically available in other cross sectional imaging 
modalities. Such an arrangement could be used, for example, to obtain a 
different perspective on a patient's spine which is typically curved, and which 
may be further curved due to a disease such as kyphosis or scoliosis. Using the 
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tilted CT slice images allows physician 18 to follow the curvature of a patient's 
spine while obtaining a single CT slice images of the spine. In the embodiment 
shown in FIGURE 6, the angular relationship between display 22 and half- 
mirror 24 is maintained. 

FIGURES 7 and 8 show embodiments of image overlay system 10 in 
which tilted CT slice images are obtained from synthetic (or re-sampled) CT 
slice images that are oblique cuts of a computer-generated 3D image or digital 
representation of some volume within patient 16. In FIGURE 7, display 22 is 
tilted at some angle 44 equal to a° to achieve a rotation of overlay image 28 
with respect to image plane 20 at an angle 45 that is also equal to cc°, but in the 
opposite direction. In FIGURE 8, half-mirror 24 is tilted so as to produce a 
different angle 23B with respect to display 22 that is equal to some angle cc°. 
The result is an angle 25B between half-mirror 24 and overlay image 28. Both 
of the embodiments shown in FIGURES 7 and 8 use a computer-generated 3D 
image to form a new cross-section that is the overlay image 28. 

FIGURES 9 and 10 show embodiments of image overlay system 10 in 
which the surgical device placement mechanism is a robot device 30A that is 
remotely controlled. Currently, there is no such small imager-compatible 
surgical robot is available commercially. Several prominent research groups 
are developing such devices that are expected to enter the marketplace in the 
near future. In the embodiment of FIGURE 9, the surgeon 18 uses a joystick 
control unit 48 that is connected to robot placement mechanism 30 A through a 
cable 56. In the embodiment of FIGURE 10, surgeon 18 uses a desktop 
computer 50 that is connected to robot placement mechanism 30A through a 
cable 57. In these embodiments, robot placement mechanism 30A can be a 
device that is, for example, electromechanical, pneumatic, ultrasonic, or some 
other control arrangement. As with the placement mechanism 30 used in the 
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embodiment of FIGURE 1, robot placement mechanism 30A is designed to 
allow surgical instrument 32 to be moved in the X, Y directions of a constrained 
plane and rotated within that same plane so that it can be properly inserted in 
patient 16. 

FIGURE 1 1 shows an embodiment of image overlay system 10 in which 
a passive mechanism 30 for the placement of a surgical device 32 is used. As 
in the embodiment of FIGURE 1, placement mechanism 30 is a mechanical 
device that is designed to allow surgical instrument 32 to be moved in the X 
and Y directions of a constrained plane and rotated within that same plane so 
that it can be properly inserted in patient 16. Placement mechanism 30 is 
positioned by surgeon 18's hand 52 into the proper location and orientation for 
insertion of surgical instrument 32 into patient 16. 

FIGURE 12 shows image overlay system 10 without placement 
mechanism 30. In the embodiment of FIGURE 12, physician 18 holds surgical 
instrument 32 in his hand 52 and positions instrument 32 for insertion into 
patient 16 using the laser marker 21 of the CT scanner 1 1 as a visual guide for 
proper placement and orientation of instrument 32 prior to insertion. 

FIGURE 13 also shows image overlay system 10 without placement 
mechanism 30. In the embodiment of FIGURE 13, physician 18 again holds 
surgical instrument 32 in his hand 52 and positions instrument 32 for insertion 
into patient 16. In this embodiment, however, the physician uses a tracking 
system that includes a base unit 54 attached to mounting device 26 and a 
transmission unit 56 attached to surgical instrument 32. Several commercially 
available tracking devices can be used for this purpose, such as Polaris 
(Northern Digital, Canada) , Flashpoint (Image Guided Technologies, Boulder, 
CO). Tracking information generated by units 54 and 56 is available to 
surgeon 18 on personal computer 50 that is connected to base unit 54 through a 
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cable 57 running between base unit 54 and computer 50. In this embodiment, 
surgeon 18 uses the tracking data available on computer 50 as a guide for 
proper positioning and orientation of instrument 32 prior to insertion in patient 
16. Typically, the position data is numerical data reflecting the coordinate 
location of the surgical instrument 32 with respect to the patient 16. 

FIGURE 14 again shows the image overlay system 10 of the present 
invention with a tracking system that includes a base unit 54 attached to 
mounting device 26 and a transmission unit 56 attached to surgical instrument 
32. In the embodiment of FIGURE 14, system 10 also includes a surgical 
device placement mechanism 30 for holding and positioning surgical 
instrument 32. Tracking information generated by units 54 and 56 is again 
available to surgeon 18 on computer 50, which is connected to base unit 54 
through cable 57. Here, surgeon 18 uses the tracking data available on 
computer 50 as a guide for positioning and orientation of placement mechanism 
30 and instrument 32 held by mechanism 30 prior to insertion in patient 16. 
Here again, the position data generated by tracking device units 54 and 56 is 
numerical data reflecting the coordinate location of the surgical instrument 32 
with respect to the patient 16. 

FIGURE 15 shows image overlay system 10 without the placement 
mechanism 30, but with tracking units 54 and 56. In the embodiment of 
FIGURE 15, physician 18 again holds surgical instrument 32 in his hand 52 
and positions instrument 32 for insertion into patient 16 using tracking base 
unit 54 attached to mounting device 26 and transmission unit 56 attached to 
surgical instrument 32. Tracking information generated by units 54 and 56 is 
again displayed on personal computer 50, but also overlaid on overlay image 
28. This is accomplished by a transfer of such information from computer 50 to 
display 22 through cable 58 connecting computer 50 to display 22. The 
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placement of the tracking data on overlay image 28 readily allows surgeon 18 
to use the tracking data as a guide for proper positioning and orientation of 
instrument 32 prior to insertion in patient 16. Here again, the position data is 
numerical data reflecting the coordinate location of the surgical instrument 32 
with respect to the patient 16. 

The embodiment of image overlay system 10 shown in FIGURE 16 again 
uses a tracking system that includes base unit 54 attached to mounting device 
26 and transmission unit 56 attached to surgical instrument 32. In the 
embodiment of FIGURE 16, there is also surgical device placement mechanism 
30 for holding and positioning surgical instrument 32. Tracking information 
generated by units 54 and 56 is again available to surgeon 18 on computer 50 
and overlaid on overlay image 28. This is accomplished by a transfer of such 
information from computer 50 to display 22 through cable 58 connecting 
computer 50 to display 22. Here again, the placement of the tracking data on 
overlay image 28 readily allows surgeon 18 to use the tracking data as a guide 
for proper positioning and orientation of placement mechanism 30 and 
instrument 32, held by mechanism 30, prior to insertion in patient 16. As in the 
prior embodiments, the position data generated by tracking device units 54 and 
56 is numerical data reflecting the coordinate location of the surgical 
instrument 32 with respect to the patient 16. 

FIGURE 17 shows an embodiment of the image overlay system of the 
present invention in which coordinate data from an encoded instrument 
placement mechanism 30B is visually superimposed on virtual overlay image 
28 to assist surgeon 1 8 in the placement of surgical device 32 in patient 16. 
Here again, encoded placement mechanism 30B constrains surgical device 32 
in a plane that coincides with the laser marker 21 of CT scanner 1 1. Placement 
mechanism 30B is also designed to allow surgical device 32 to be moved in the 
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X, Y directions of the constrained plane and rotate within the same plane so that 
device 32 can be properly inserted in patient 16. In this embodiment of 
placement mechanism 3 OB, there is also included an encoding mechanism for 
providing the X, Y coordinates of device 32 and coordinates corresponding to 
its rotational orientation. Encoded instrument placement mechanism 30B can 
be a passive placement device or robot of some sort, not specified further here. 
The coordinate data from encoded instrument placement mechanism 30B is sent 
to personal computer 50, and then transmitted to display 22 so that it can be 
visually superimposed on virtual overlay image 28. 

FIGURE 18 shows an embodiment of image overlay system 10 in which 
slice images 60 from CT scanner 1 1 are registered with multi-modal pre- 
operative slice images 62 and 64 from, for example, MRI and PET scanners, 
respectively, so that a composite, more comprehensive slice image is sent to 
display 22 for overlay image 28C . Composite overlay images are desirable 
because different types of scans have different advantages. For example, CT 
scans are good for hard tissue, such as bones and ligaments. In contrast, soft 
tissue differentiation in CT scans is relatively poor, whereas it is good in MRI 
scans. Typically, the auxiliary scans are done pre-operatively, then spatially 
registered to the coordinate space of the intra-operative CT images, by some 
standard means medical image registration not specified further in this 
document. FIGURES 19, 20 and 21 show a robust calibration procedure 
without the use of a calibration object. Step 1 of the calibration procedure is 
shown in FIGURE 19. Step 1 is to confirm that display 22 is horizontal, CT 
gantry 12 is vertical, and half-mirror 24 is oriented at 45° with respect to image 
plane 20. Preferably, a bubble level 66, as shown in FIGURE 19, can be used 
for this purpose. 
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Step 2 of the calibration procedure, which is shown in FIGURE 20, is to 
translate monitor 22 or half-mirror 24 off-plane to align the current image plane 
with CT scanner 1 l*s laser marker 21 . Preferably, a CT slice image of patient 
16 is taken and the height of display 22 is adjusted some distance d until the 
plane of overlay image 28 coincides with the plane of the axial laser marker 21 
of scanner 1 1 . 

Step 3 of the image overlay calibration procedure can be explained in 
FIGURE 21. In step 3, display 22 is translated in plane, or the image on display 
22 is translated, until the overlay image 28 is aligned with patient 16*s body. 
Preferably, the contour of patient 16's body is highlighted in the image on 
display 22 using image processing. Alternatively, fiducial marks (not shown) 
can be applied to the body of patient 16 prior to taking CT images. . The 
fiducials are then highlighted using the imaging processor of display 22. 
Device placement mechanism 30 may also be instrumented with similar 
fiducials. Display 22 is then adjusted within plane motion or the image on 
display 22 is adjusted with panning via software control without moving 
display 22 until overlay image 28 matches the contour of patient 16's body and 
all selected fiducials and anatomical landmarks. FIGURE 22 shows passive 
surgical device placement mechanism calibration. In this procedure, CT gantry 
12 is brought to a vertical position. Bubble level 66 (as shown in Figure 19) is 
used to confirm and insure the vertical position of CT gantry 12. The laser 
light of scanner lis laser marker 21 in the axial image plane must be cast in the 
central line of surgical device placement mechanism 30. FIGURE 23 shows a 
calibration method applicable to encoded surgical device placement 
mechanism, either passive or active robotic mechanism. The method employs 
rigid body fiducial based stereotactic registration system attached to the 
surgical device placement mechanism. The rigid fiducial frame is assembled 
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from several rods, in a known geometrical pattern. Upon imaging, the rods are 
cut across the CT slice and the mark of each rod is detected in the CT image. 
Based on the relative position of these marks and the a priori known relative 
spatial position of the rods, one can calculate the spatial location of the fiducial 
frame with respect to the CT image. This completes the registration process. 

Although the present invention has been described in terms of particular 
embodiments, it is not intended that the invention be limited to those 
embodiments. Modifications of the disclosed embodiments within the spirit of 
the invention will be apparent to those skilled in the art. The scope of the 
present invention is defined by the claims that follow. 
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WHAT IS CLAIMED IS: 

1. An image overlay system to assist surgical interventions 
by a surgeon on a patient inside a cross-sectional imaging system 
including an imaging gantry for taking slice images of the patient, 
the image overlay system comprising: 

a half-mirror positioned between the surgeon and the 
patient, 

a display positioned with respect to the half-mirror so that 
slice images shown on the display are projected onto the half- 
mirror so that the slice images appear to be floating inside the 
patient in correct size and position, and 

a holding attachment connected to the imaging gantry for 
supporting the half-mirror and display. 

2. The image overlay system as recited in claim 1, further 
comprising a mechanism for placement of a surgical device with 
respect to the patient. 

3. The image overlay system as recited in claim 1, wherein 
the holding attachment supporting the display allows the display to 
be off-plane translated so that multiple slice images can be 
displayed with respect to the patient at the respective locations at 
which the slice images were taken by the cross-sectional imaging 
system. 

4. The image overlay system as recited in claim 1, wherein 
the holding attachment supporting the half-mirror allows the half- 
mirror to be translated off-plane so that multiple slice images can 
be displayed with respect to the patient at the respective locations 
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at which the slice images were taken by the cross-sectional 
imaging system. 

5. The image overlay system as recited in claim 1, wherein 
the holding attachment supporting the display allows the display to 
be in-plane translated so that an image could be panned inside the 
plane of imaging, at which the slice images were taken by the 
cross-sectional imaging system. 

6. The image overlay system as recited in claim 1 further 
comprising a computer connected to the display, wherein the 
computer is programmed to allow the display's image to be 
horizontally translated so that an image could be panned inside the 
plane of imaging at which the slice images were taken by the 
cross-sectional imaging system. 

7. The image overlay system as recited in claim 1, wherein 
the holding attachment supporting the display and half-mirror 
includes a connection that allows the half-mirror and display to be 
simultaneously tilted to avoid interference with the patient while 
in the cross-sectional imaging system. 

8. The image overlay system as recited in claim 7, wherein 
the connection is a rotational joint. 

9. The image overlay system as recited in claim 1, wherein 
the imaging gantry is capable of being tilted so that tilted slice 
images of the patient can be obtained and displayed using the 
image overlay system. 

10. The image overlay system as recited in claim 1, wherein 
the holding attachment supporting the display allows the display to 
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be tilted so that tilted slice images can be displayed with respect to 
the patient. 

1 1. The image overlay system as recited in claim 1 further 
comprising a computer connected to the display, wherein tilted 
slice images are formed by the computer using a three-dimensional 
image of the patient formed from a plurality of slice images of the 
patient taken by the cross-sectional imaging system. 

12. The image overlay system as recited in claim 1, wherein 
the holding attachment supporting the half-mirror allows the half- 
mirror to be tilted so that tilted slice images can be displayed with 
respect to the patient. 

13. The image overlay system as recited in claim 12 further 
comprising a computer connected to the display, wherein tilted 
slice images are formed by the computer using a three-dimensional 
image of the patient formed from a plurality of slice images of the 
patient taken by the cross-sectional imaging system. 

14. The image overlay system as recited in claim 2, wherein 
the placement mechanism is a robot mechanism, and wherein the 
image overlay system further comprises a joystick for controlling 
the robot placement mechanism. 

15. The image overlay system as recited in claim 2, wherein 
the placement mechanism is a robot mechanism, and wherein the 
image overlay system further comprises a computer for controlling 
the robot placement mechanism. 

16. The image overlay system as recited in claim 2, wherein 
the placement mechanism is a passive mechanical mechanism 
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manually actuated by the surgeon. 

17. The image overlay system as recited in claim 16, 
wherein the placement mechanism allows the surgical device to be 
moved in X and Y directions of a constrained plane and rotated 
within the plane, whereby the surgical instrument can be manually 
positioned and oriented by the surgeon with respect to the patient. 

18. The image overlay system as recited in claim 2, wherein 
the placement mechanism allows the surgical device to be moved 
in X and Y directions of a constrained plane and rotated within the 
plane, whereby the surgical instrument can be manually positioned 
and oriented by the surgeon with respect to the patient. 

19. The image overlay system as recited in claim 1, wherein 
the cross-sectional imaging system includes a laser marker that is 
used by the surgeon to position a surgical device with respect to 
the patient free-hand. 

20. The image overlay system as recited in claim 1 further 
comprising a tracking device for providing tracking data that is 
used by the surgeon to position a surgical device with respect to 
the patient. 

21. The image overlay system as recited in claim 20, 
wherein the surgeon positions the surgical device with respect to 
the patient free-hand, 

22. The image overlay system as recited in claim 20, 
wherein the tracking device includes a base unit mounted on the 
holding attachment and a transmitting unit mounted on the 
surgical device. 
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23. The image overlay system as recited in claim 2 further 
comprising a tracking device for providing tracking data that is 
used by the surgeon to position a surgical device with respect to 
the patient. 

24. The image overlay system as recited in claim 23, 
wherein the tracking data is used by the surgeon to position the 
placement mechanism and thereby the surgical device with respect 
to the patient. 

25. The image overlay system as recited in claim 23, 
wherein the tracking device includes a base unit mounted on the 
holding attachment and a transmitting unit mounted on the 
surgical device. 

26. The image overlay system as recited in claim 20 further 
comprising a computer for visually superimposing on the slice 
images shown on the display the tracking data obtained from the 
tracking device to assist the surgeon in positioning the surgical 
device with respect to the patient free-hand. 

27. The image overlay system as recited in claim 23 further 
comprising a computer for visually superimposing on the slice 
images shown on the display the tracking data obtained from the 
tracking device to assist the surgeon in positioning the placement 
mechanism and thereby the surgical device with respect to the 
patient. 

28. The image overlay system recited in claim 1, wherein 
the cross-sectional imaging system is a computed tomography 
imaging system. 
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29. The image overlay system recited in claim 28, wherein 
the cross-sectional imaging system further comprises a computer 
for registering slice images taken by the computed tomography 
imaging system with slice images from cross-sectional imaging 
systems of a type different from the computed tomography 
imaging system. 

30. The image overlay system recited in claim 29, wherein 
the different type of imaging system is a magnetic resonance 
imaging system. 

31. The image overlay system recited in claim 29, wherein 
the different type of imaging system is a position emission 
tomography imaging system. 

32. The image overlay system recited in claim 29, wherein 
the different type of imaging system is a single photon emission 
computed tomography imaging system. 

33. The image overlay system as recited in claim 1, wherein 
the holding attachment includes a plurality of arms and rotational 
joints that allow a plurality of degrees-of-freedom in movement 
and positioning of the holding attachment. 

34. The image overlay system as recited in claim 33, 
wherein the holding attachment has seven degrees-of-freedom. 

35. An image overlay system for use by a physician with a 
patient inside a computed tomography ("CT") scanning system for 
taking slice images of the patient inside a CT gantry, the image 
overlay system comprising: 

a half-mirror position between the physician and the patient, 
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an overlay display positioned with respect to the half-mirror 
so that slice images shown on the overlay display are projected 
onto the half-mirror, and 

a holding attachment supporting the half-mirror and display 
and being connected to the CT gantry. 

36. The image overlay system as recited in claim 35 further 
comprising a mechanism for placement by the physician of a 
surgical device with respect to the patient. 

37. The image overlay system as recited in claim 35, 
wherein the holding attachment allows the overlay display to be 
off-plane translated and/or the half-mirror to be off-plane 
translated so that multiple slice images can be displayed with 
respect to the patient at the respective locations at which the slice 
images were taken by the CT scanning system. Do not we miss 
here one? In-plane motion of the display to achieve panning? (See 
claim 5.) 

38. The image overlay system as recited in claim 35, 
wherein the holding attachment supporting the display allows the 
display to be in-plane translated so that an image could be panned 
inside the plane of imaging, at which the slice images were taken 
by the cross-sectional imaging system. 

„ 39. The image overlay system as recited in claim 35, 
wherein the holding attachment allows the half -mirror to be tilted 
to avoid interference with the patient while in the CT scanning 
system or to allow tilted slice images to be displayed with respect 
to the patient. 
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40. The image overlay system as recited in claim 35, 
wherein the CT gantry is capable of being tilted so that tilted slice 
images of the patient can be obtained and displayed with respect to 
the patient. 

41. The image overlay system as recited in claim 35 further 
comprising a computer that is programmed to allow the overlay 
display's image to be horizontally translated an image could be 
panned inside the plane of imaging at which the slice images were 
taken by the cross-sectional imaging system. 

42. The image overlay system as recited in claim 36, 
wherein the placement mechanism is a robot mechanism, and 
wherein the image overlay system further comprises a joystick for 
controlling the robot placement mechanism. 

43. The image overlay system as recited in claim 36, 
wherein the placement mechanism is a robot mechanism, and 
wherein the image overlay system further comprises a computer 
for controlling the robot placement mechanism. 

44. The image overlay system as recited in claim 36, 
wherein the placement mechanism is a mechanical mechanism 
manually actuated by the physician. 

45. The image overlay system as recited in claim 44, 
wherein the placement mechanism allows the surgical device to be 
moved in X and Y directions of a constrained plane and rotated 
within the plane so that the surgical instrument can be positioned 
and oriented with respect to the patient. 

46. The image overlay system as recited in claim 44, 
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wherein the CT scanning system includes a laser marker that is 
used by the physician to position the surgical device with respect 
to the patient free-hand. 

47. The image overlay system as recited in claim 35 further 
comprising a tracking device for providing tracking data that is 
used by the physician to position a surgical device with respect to 
the patient. 

48. The image overlay system as recited in claim 36 further 
comprising a tracking device for providing tracking data that is 
used by the physician to position the placement mechanism and 
thereby the surgical device with respect to the patient. 

49. The image overlay system as recited in claim 35 further 
comprising a tracking device for providing data that is used to 
position a surgical device with respect to the patient and a 
computer for visually superimposing on the slice images shown on 
the overlay display tracking data obtained from the tracking device 
to assist in the placement of the surgical device. 

50. The image overlay system as recited in claim 35 further 
comprising a computer for registering slice images taken by the 
computed tomography scanning system with slice images from 
cross-sectional imaging systems of a type different from the 
computed tomography scanning system. 

5 1 . The image overlay system as recited in claim 35 further 
comprising a calibration object for use in registering the half- 
mirror, overlay display and the CT scanning system. 

52. The image overlay system as recited in claim 51, 
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wherein the calibration object is a three-dimensional prism. 

53. The image overlay system as recited in claim 1 further 
comprising a calibration object for use in registering the half- 
mirror, overlay display and the cross-sectional imaging system. 

54. The image overlay system as recited in claim 53, 
wherein the calibration object is a three-dimensional prism 

55. The image overlay system as recited in claim 52 
wherein the three-dimensional prism is triangularly-shaped. 

56. The image overlay system as recited in claim 54 
wherein the three-dimensional prism is triangularly-shaped. 

57. A method for registering the image overlay system 
recited in claim 35 comprising the steps of: 

placing on the CT gantry a fiducial marker, 

adjusting the calibration object to a laser marker of the CT 

scanning system, 

obtaining a slice image of the fiducial marker, 
precalibrating the overlay display and displaying on the 

overlay display and adjusting to full size the slice image of the 

fiducial marker, 

while looking through the half-mirror, adjusting the slice 

image of the calibration object to the actual calibration object on 

the CT gantry. 

58. The method recited in claim 57, wherein the calibration 
object is a three-dimensional prism. 

59. A method for registering the image overlay system 
recited in claim 1 comprising the steps of: 
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placing on the imaging gantry a fiducial marker, 
adjusting the calibration object to a laser marker of the 

cross-sectional imaging system, 

obtaining a slice image of the fiducial marker, 
precalibrating the display and displaying on the display and 

adjusting to full size the slice image of the fiducial marker, 

while looking through the half-mirror, adjusting the slice 

image of the calibration object to the actual calibration object on 

the imaging gantry. 

60. The method recited in claim 59, wherein the calibration 
object is a three-dimensional prism. 

61. A method for registering the image overlay system 
recited in claim 1 comprising the steps of: 

confirming that the display is horizontal, the imaging gantry 
is vertical, and the half-mirror is oriented at a selected angle with 
respect to an image plane of the cross-sectional imaging system, 

obtaining a slice image of a patient on the imaging gantry 
and adjusting the height of the display until the plane of the 
overlay image coincides with the plane of an axial laser marker of 
the cross-sectional scanner, 

translating the display in plane or the slice image displayed 
on the display until the slice image is aligned with the patient's 
body. 

62. The method as recited in claim 61, wherein the step of 
confirming that the display is horizontal, the imaging gantry 
vertical, and the half-mirror oriented at a selected angle is 
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performed using a bubble level. 

63. The method as recited in claim 61, wherein the selected 
angle is 45 s . 

64. A method of calibrating the surgical device placement 
mechanism recited in claim 2 comprising the steps of: 

bringing the imaging gantry to a vertical position, and 
ensuring that a laser marker of the cross-sectional imaging 

system is in a central line of the surgical device placement 

mechanism. 

65. The method of claim 64 further comprising the step of: 
confirming the vertical position of the imaging gantry using 

a bubble level. 

66. A method for performing a surgical intervention on a 
patient inside a cross-sectional imaging scanner system including 
an imaging gantry, the method comprising the steps of: 

obtaining a slice image of the patient using the scanner 
system, 

displaying the slice image on an overlay display, 
projecting the slice image displayed on the display onto a 

half-mirror so that the slice image is reflected from the half-mirror, 
viewing the patient through the half-mirror with the slice 

image reflected therefrom, whereby the slice image appears to be 

overlaying and floating inside the patient's body in correct size and 

position, 

using the overlaying slice image to position a surgical 
device with respect to the patient, and 
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performing the surgical intervention using the positioned 
surgical device. 

67. The method as recited in claim 66 further comprising 
the step of positioning the surgical device with respect to the 
patient using a placement mechanism that holds the surgical 
device 

68. The method as recited in claim 66 further comprising 
the step of translating the display off-plane so that multiple slice 
images can be displayed with respect to the patient at the 
respective locations at which the slice images were taken. 

69. The method as recited in claim 66 further comprising 
the step of translating the half-mirror off-plane so that multiple 
slice images can be displayed with respect to the patient at the 
respective locations at which the slice images were taken. 

70. The method as recited in claim 66 further comprising the 
step of translating the display in-plane so that an image could be 
panned inside the plane of imaging at which the slice images were 
taken. 

71. The method as recited in claim 66 further comprising 
the step of translating the slice image displayed on the display so 
an image could be panned inside the plane of imaging at which the 
slice images were taken. 

72. The method as recited in claim 66 further comprising 
the step of simultaneously tilting the half-mirror and display to 
avoid interference with the patient while the patient is in the 
scanner system. 
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73. The method as recited in claim 66 further comprising 
the step of tilting the imaging gantry so that tilted slice images of 
the patient can be obtained and displayed. 

74. The method as recited in claim 66 further comprising 
the step of generating tilted slice images using a computer- 
generated three-dimensional image of the patient formed from a 
plurality of slice images of the patient taken by the scanner 
system. 

75. The method as recited in claim 66 further comprising 
the step of tilting the half-mirror so that tilted slice images can be 
displayed with respect to the patient. 

76. The method as recited in claim 67 further comprising 
the step of positioning the surgical device with respect to the 
patient using a robot placement mechanism that is controlled by a 
joystick. 

77. The method as recited in claim 67 further comprising 
the step of positioning the surgical device with respect to the 
patient using a robot placement mechanism that is controlled by a 
personal computer. 

78. The method as recited in claim 66 further comprising 
the step of placing a surgical device with respect to the patient 
free-hand. 

79. The method as recited in claim 66 further comprising 
the step of placing a surgical device with respect to the patient 
using tracking data that is obtained from a tracking device. 

80. The method as recited in claim 79, wherein the tracking 
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data is numerical data. 

81. The method as recited in claim 80, wherein the tracking 
data is visually displayed on the slice image reflected from the 
half-mirror. 

82. The method as recited in claim 66, wherein the 
scanning system is a computed tomography scanning system. 

83. The method as. recited in claim 66, wherein the 
scanning system is a magnetic resonance imaging system. 

84. The method as recited in claim 66, wherein the 
scanning system is a position emission tomography scanning 
system. 

85. The method as recited in claim 66, wherein the 
scanning system is a single photon emission computed 
tomography scanning system. 
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